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ABSTRACT 



Centralised targeting registries are increasingly used to allocate social assistance benefits in developing 
countries. This paper provides the first attempt to identify the relative importance of two key design 
issues for targeting accuracy: (1) which households to survey for inclusion in the targeting registry and 
(2) how to rank surveyed households. We evaluate the performance of Indonesia's Unified Database 
for Social Protection Programmes (UDB), the largest targeting registry in the world, which is used 
to provide social assistance to more than 25 million households. Linking administrative data with an 
independent household survey, we find that the UDB system is more progressive than previous targeting 
approaches used in Indonesia, leading to a decrease in benefit leakage to non-poor households. However, 
if poor households are not surveyed in the first place, even a perfect ranking method cannot prevent 
their exclusion. Under a simulation that considers enumerating and estimating proxy-means testing 
(PMT) scores for all households (as in a census), we estimate a one-third decrease in undercoverage 
compared to focusing on households that have been registered in the UDB. Investigating household- 
and community-level correlates of misenumeration and misclassification, we find evidence that local 
communities use different definitions of poverty and have better information on the welfare status of 
their members. 
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1. Introduction 



Social assistance programmes are currently estimated to cover as many as one billion people in the 
developing world (International Labour Organization 2010). These programmes are often targeted 
to the neediest population groups, which are identified based on socioeconomic status, in order to 
maximise their effectiveness in improving social welfare given a limited budget. However, identifying 
and reaching the intended beneficiaries can be challenging, especially in developing countries where a 
large part of the population works in the informal sector and official income registries do not exist. In the 
past 20 years, low- and middle-income countries have increasingly used centralised targeting registries 
to select recipients of social assistance programmes. 2 For such registries, basic household and individual 
information is typically collected for a subset of the population that is considered potentially eligible 
for social assistance (because conducting a full census of all households is usually cost prohibitive). 3 
This information is then used to determine eligibility, most commonly based on proxy-means testing 
(PMT). 4 

This paper deals with two key challenges that arise in the development of any targeting registry. The first 
challenge is how to identify households for inclusion in the registry or, in other words, who to survey 
within the entire population. Properly addressing this issue is essential to ensuring that poor households 
are included in the registry in the first place, thereby avoiding what we refer to here as 'misenumeration' 
errors. The second challenge is how to assess the eligibility of those surveyed, that is, how to estimate 
their socioeconomic status in order to rank or classify them. The main concern in this step is to minimise 
what are referred to here as 'misclassification' errors that stem from the poor households surveyed being 
deemed ineligible and from non-poor (surveyed) households being wrongly classified as poor. 

Misenumeration and misclassification have strong implications for targeting accuracy, which is 
commonly assessed using two key measures: leakage (or 'inclusion error'), when non-intended 
beneficiaries receive programme benefits, and undercoverage (or 'exclusion error'), when intended 
beneficiaries do not receive programme benefits (Cornia and Stewart 1995). Many programmes and 
countries today suffer from the adverse consequences of inaccurate targeting (Acosta et al. 2011). 
To date, however, little is known about the relative importance of household registration and ranking 
in determining the accuracy of targeting registries, as most existing studies focus on errors due to 
misclassification. 5 



2 Examples of countries using centralised targeting registries besides Indonesia include Brazil, Chile, Colombia, India, 
Mexico, and Philippines (see, for example, Castaneda et al. (2005) for a review of the experience of Latin American countries; 
Dreze and Khera (2010) for the Indian Below-Poverty Line Census). 

3 Compared with common population census questionnaires, targeting registry questionnaires collect more detailed 
socioeconomic information at household and individual levels. They are therefore generally administered to a subset of the 
population, rather than the full population, in order to limit costs. 

4 PMT scores are constructed based on simple socioeconomic indicators that are relatively easy to collect and less prone to 
misreporting than expenditures or income. These indicators are combined into a single measure of welfare using weights 
typically derived from consumption regressions estimated from an auxiliary survey. Using these predicted measures of welfare 
can be a cost-effective way to identify beneficiaries of social programmes to the extent that they are sufficiently accurate. 

5 This includes evaluations of targeting effectiveness (e.g., Skoufias et al. 2001; Banerjee et al. 2007), as well as most of the 
optimal targeting literature, which can be divided between studies comparing different targeting methods and studies focusing 
on the design of PMT formulas for ranking. Studies comparing the relative advantages of different targeting methods (e.g., 
Coady et al. 2004, Banerjee et al. 2007, Alatas et al. 2012 and 2013a, and Karlan and Thuysbaert 2013) find that no method 
clearly dominates in terms of commonly used objective performance measures. However, in general, the evidence suggests 
that community targeting is best for identifying the very poorest households. Other studies focusing on the design of optimal 
PMT formulas (e.g., Sumarto et al. 2007, Muller and Bibi 2010, and Bah 2013) show that targeting errors are unavoidable 
when using simple indicators, although the degree of error can be minimised with more careful selection of the proxies for 



This paper provides the first attempt to assess the relative contribution of the household registration and 
ranking processes to the overall accuracy of a centralised targeting registry. We do so using Indonesia's 
newly developed household targeting registry and aim to identify priority actions for improving 
targeting effectiveness. Established in 2012, the Unified Database (UDB) for Social Protection 
Programmes is intended to cover the poorest 40% of the Indonesian population. 6 More than 25 million 
households have been registered in the UDB using an innovative approach based on a pre-listing of 
households to be surveyed, constructed through census-based poverty mapping (Elbers et al. 2003) and 
complemented with suggestions from local communities. These households were subsequently ranked 
by their predicted welfare, estimated using district-specific 7 PMT formulas. The UDB has been used to 
deliver more than US$4 billion annually (Rp 43 trillion) in central government social assistance (based 
on 2013 figures). 8 This includes the largest social assistance programmes in the country: a rice subsidy 
programme known as Rice for the Poor (Subsidi Beras Bagi Masyarakat Berpendapatan Rendah 
or Raskin), a health insurance programme known as Public Health Insurance (Jaminan Kesehatan 
Masyarakat or Jamkesmas), and an Unconditional Cash Transfer programme {Bantuan Langsung Tunai 
or BLT). Before UDB's establishment, beneficiaries of these programmes were selected using ad hoc 
targeting approaches. 

Our analysis for this paper proceeds in three steps. First, we evaluate the targeting performance of the 
UDB against the performance of past approaches to beneficiary selection used for the three main social 
assistance programmes. We use data from an independent survey known as the Indonesian Household 
Socioeconomic Survey (Survei Sosial Ekonomi Rumah Tangga or Suseti), which were matched with 
UDB administrative data. Suseti contains information on household expenditures per capita, which is 
not in the UDB, as well as information on the receipt of Raskin, Jamkesmas, and BLT at baseline (i.e., 
before the establishment of the UDB). 

We find that targeting using the UDB is more progressive than previous approaches to beneficiary 
selection. In particular, the UDB leads to a substantial reduction in leakage of benefits to non-poor 
households. This decrease in leakage is largest for Raskin, for which the proportion of the richest 60% 
of households receiving benefits is expected to fall from nearly 75% to 25%. Our findings highlight the 
trade-offs between undercoverage and leakage found in many studies of targeting effectiveness (Grosh 
and Baker 1995). There are, indeed, more limited improvements in terms of undercoverage, which can 
be due to both misenumeration and misclassification errors. 

Second, we differentiate the contribution of the enumeration and the PMT-based ranking processes 
to targeting errors, and in particular undercoverage. Through an assessment of the counterfactual 
performance that would be observed if all households had been enumerated (as in a census), we find 
evidence of enumeration gaps in the UDB that lead to undercoverage of poor households. Under this 
hypothetical scenario, severe undercoverage of all programmes falls by about one-third relative to a 



consumption. One notable exception is Alatas et al. (2013b), which shows that self-targeting has the potential to reduce 
misenumeration errors at the registration stage. 

6 References to 'poor' and 'non-poor' in relation to the UDB are meant to distinguish on the one hand households in the bottom 
40% which is the target coverage of the UDB, and on the other hand those in the upper 60% of the consumption distribution 
that are meant to be excluded from the UDB. 

7 Indonesia's administrative divisions proceed from province to district to subdistrict to village to hamlet. There were 497 
districts at the time of the establishment of the UDB. 

8 In June 2013, the Government of Indonesia announced a reduction in fuel subsidies, accompanied by a set of four 
compensation programmes to mitigate the reduction's effect on poor and vulnerable households. 



targeting system based only on those households actually included in the UDB. Depending on the 
social planner's welfare function (i.e., the relative weights on the poorest households in the population), 
our findings suggest large gains from reallocating scarce administrative resources towards increasing 
survey coverage to minimise undercoverage of poor households in the UDB. In particular, we show that 
increased enumeration costs to cover the full population would amount to about 11% of the value of 
additional benefits that would be received annually by households from the poorest 30%. In other words, 
there should be a stronger focus on ensuring that an adequate number of households are surveyed. If 
poor households are not enumerated in the first place, even a perfect PMT algorithm cannot prevent 
their exclusion. 

Third, we identify household- and community-level correlates of misenumeration and misclassification. 
Ownership of household assets that are difficult to observe and are not recorded in the UDB is associated 
with a lower probability of registration in the UDB. However, for households that are nevertheless 
registered, ownership of such assets is associated with a greater potential of being misclassified as poor. 
Our findings also suggest some form of strategic interaction with other social programmes during the 
UDB registration process. For example, receipt by households of informal support from (religious) 
nongovernmental organisations is associated with a higher likelihood of registration in the UDB. 
Furthermore, we find mixed evidence on the role of local (neighbourhood) officials in the targeting 
process. Households residing in villages with elected neighbourhood heads are more prone to exclusion 
errors. This may be due to 'voter capture' in which preferential treatment in the targeting process 
may be given to those residents who participate in neighbourhood activities including local elections 
(Kurasawa 2009). 

Our paper contributes to the literature in public and development economics on optimal targeting of 
social programmes. Most studies use a single survey to identify intended and actual recipients, that 
is, who is poor and who is receiving government benefits. However, relying solely on household self- 
reporting on whether they receive benefits does not allow for a full understanding of what happens at the 
multiple stages of the targeting process before benefits are delivered to households. To our knowledge, 
this paper provides the first analysis based on actual administrative data on household eligibility for 
government social programmes linked with data on household expenditures from an independent 
survey. Using administrative data allows us to identify the relative contribution to overall targeting 
accuracy of (1) the decision on which potentially poor households to survey and (2) the estimation 
of their socioeconomic status based on the data collected. As a result, we are able to prioritise policy 
options to minimise the potential exclusion of the poorest households from increasingly used targeting 
registries of the sort we study in Indonesia. 

Our findings have important implications for ongoing policy debates in developing countries concerning 
the design of efficient and equitable targeting registries. Overall, our results provide further evidence 
on the difficulty of accurate targeting in countries like Indonesia that have considerable clustering 
of households around the poverty line. Nevertheless, the research design allows us to clarify how 
improvements in the enumeration process can lead to large gains in overall targeting effectiveness. 

The remainder of the paper is organised as follows. Section 2 provides background information on 
Indonesia's UDB. Section 3 presents the Suseti survey and its features. Section 4 assesses the predicted 
targeting accuracy of the UDB. Section 5 explores the determinants of UDB accuracy. Section 6 
concludes with policy recommendations. 



2. The Unified Database for Social Protection Programmes 



In this section, we describe the two main steps in establishing a centralised targeting registry of 25 
million households ranked according to their socioeconomic status: data collection (enumeration) 
and PMT modelling (ranking). 9 First, the data collection stage involved pre-identifying all potentially 
eligible households that should be surveyed. Given the lack of accurate pre-existing data on which 
households are the poorest, the government adopted a new approach combining administrative data 
from the 2010 Population Census and input from local communities. 10 Second, the PMT modelling 
stage entailed incorporating proxies for consumption-based welfare measures and accounting for the 
socioeconomic diversity across regions. 

2.1 Data Collection 

The establishment of the UDB was motivated by evidence that inaccurate targeting of Indonesia's main 
social protection programmes was a major obstacle to the effectiveness of the national poverty reduction 
strategy. Previous studies revealed that these programmes suffered from significant undercoverage of 
poor households and leakage to non-poor households (e.g., World Bank 2012). The targeting errors were 
believed to be due largely to coverage and quality gaps in the previous censuses of the poor used to 
identify beneficiaries of the BLT programmes implemented in 2005 and 2008. 11 Households surveyed in 
these data collection efforts were identified based mainly on subjective consultation of enumerators from 
Statistics Indonesia {Badan Pusat Statistik or BPS) with village leaders (see, for example, SMERU 2006). 

The UDB was intended to cover a greater number of households and to avoid relying exclusively on 
subjective nominations from community leaders. The registration of households in the UDB followed 
a two-step approach: first, a 'pre-listing' of households to be surveyed produced through a poverty 
mapping exercise and, second, incorporation of suggestions from the community in the field to amend 
and complete the survey pre-listing. 

The first step was intended to mitigate the undercoverage that had plagued previous data collection 
efforts in 2005 and 2008 and to ensure that a sufficient number of households were surveyed. A poverty 
mapping exercise was conducted using the Elbers et al. (2003) methodology and the 2010 Population 
Census to estimate household welfare (approximated by per capita consumption) for the entire population. 
Target enumeration quotas were estimated using district-specific consumption-based poverty lines from 
the 2010 National Social and Economic Survey (Survei Sosial dan Ekonomi Nasional or Susenas) to 
account for income differences across Indonesia's 497 districts. 12 All households in each village with 
a predicted per capita consumption level below the enumeration quota cut-off were included on a pre- 
listing (by name and address) to be surveyed for inclusion in the UDB. 



9 Detailed information on the full process followed in establishing the UDB is available in TNP2K (2014). 

10 Alternative approaches include surveying households that request it or conducting a census in the poorest areas (e.g., 
Skoufias et al. 2001; Camacho and Conover 2011; and Karlan and Thuysbaert 2013). 

11 These two cash transfer programmes were designed to provide temporary compensation to protect poor households against 
the shocks associated with fuel subsidy reductions. See Bazzi et al. (2014) for an evaluation of the 2005 programme's impact 
on household consumption. In 2013 the BLT programme was renamed Short-Term Unconditional Cash Transfers (Bantuan 
Langsung Sementara Masyarakat or BLSM). For simplicity and because the programme is still often referred to by its original 
name, this paper uses 'BLT' to refer to both the previous and newer variants of this programme. 

12 Administered to a sample of households representative at the district level, Susenas includes a detailed consumption module 
that is used to estimate poverty lines. 
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In the second step, suggestions from communities were incorporated during the enumeration in the 
field by BPS staff responsible for registration of households on the pre-listings. In nearly all districts, 
enumerators and/or community leaders removed from the survey pre-listings those households that 
were considered non-poor or could not be found (e.g., due to relocation or death). The guidelines for 
enumerators also stipulated that households that were not on the pre-listings could be registered if 
the household (1) 'appeared poor' to the enumerators or (2) was designated as poor by other poor 
households in the community. 

The initial budget allowed for coverage of 50% of the Indonesian population. In practice, only 43% 
of all households were surveyed nationally, with varying coverage across districts. This was lower 
than expected and can be traced to the second step in the data collection process. Some households on 
the pre-listing were not actually surveyed in the field (e.g., because they were considered non-poor), 
which may have resulted in exclusion of some households. In addition, a limited number of households 
were added to the pre-listings due to reluctance among enumerators and community leaders for several 
reasons (SMERU 20 12). 13 

2.2 PMT Modelling 

The UDB registration survey collected household-level information such as demographics, housing 
characteristics, sanitation, access to basic domestic energy services, and asset ownership, along with 
information on individual household members including age, gender, schooling, and occupation. Using 
this information, households were ranked by their predicted welfare following a PMT approach. PMT 
formulas were constructed based on district-specific consumption regressions to account explicitly for 
heterogeneity across regions. 

Although the PMT approach can be a cost-effective means of identifying beneficiaries of social 
programmes in the absence of an up-to-date household registry with reliable income data, it is also prone 
to errors (e.g., Grosh and Baker 1995). In particular, targeting errors may occur due to weak predictive 
performance of the consumption models within the estimation sample (e.g., due to constraints on the set 
of socioeconomic variables available for use in the PMT regressions). Furthermore, overfilling, which 
is more likely to occur when a large number of predictors are included in the models and/or when the 
estimation is based on a small sample, may limit the validity and precision of the PMT formulas outside 
the estimation sample. 

In the remainder of the paper, we investigate the overall accuracy of Indonesia's targeting database of 
25 million households established through the data collection and PMT classification stages described 
above. 



13 In some communities, local leaders were reluctant to survey a high number of households, particularly households not 
considered to be poor. Their main concerns related to building household expectations about receiving programme benefits 
by surveying many households, when such expectations might later be disappointed (SMERU 2012). In addition, there was 
limited understanding that being surveyed would not automatically result in being selected for programmes, and local leaders 
may have feared that surveying non-poor households would increase the likelihood that these households would be selected 
to receive benefits for which they are not eligible. Similar issues had created social unrest in several communities in the past, 
especially during implementation of the 2005 census of the poor for the first BLT programme (SMERU 2006). It should also be 
noted that surveyors were paid a fixed monthly salary, rather than being paid per household surveyed, which may have reduced 
incentives to achieve greater coverage of households. 



3. The Indonesian Household Socioeconomic Survey 



To assess the accuracy of the UDB, we use data from the Indonesian Household Socioeconomic Survey 
or Suseti, which was collected by an independent survey firm and contains detailed information on 
household living conditions. 

3.1 The Suseti and its Link with the UDB 

The Suseti sample comprises 5,682 households 14 located in 600 villages spread across 6 districts in the 
provinces of Central Java (Pemalang and Wonogiri districts), Lampung (Bandar Lampung and Central 
Lampung districts), and South Sumatra (Ogan Komering Ilir and Palembang districts). The provinces 
were selected to represent a wide range of Indonesia's diverse cultural and economic geography, and 
the six districts were selected within areas where the Indonesian Conditional Cash Transfer Programme 
for Families (Program Keluarga Harapan or PKH) was to expand in 20 11. 15 In one randomly selected 
hamlet/neighbourhood {rukun tetangga or RT/ rukun warga or RW) within each of the 600 villages, 
the Suseti questionnaire was administered to nine households randomly selected from among those that 
met the PKH demographic eligibility criteria of having an expectant mother or at least one child under 
the age of 16 years old. 16 A longer version of the same questionnaire was also used to collect data from 
each neighbourhood head. 

Suseti comprises a baseline collected in March 2011, as well as an end line following the same 
households and collected in February 2012. Given the purposes of this study, we use the baseline data 
because it includes a more comprehensive set of socioeconomic variables and because the survey was 
administered closer to the July-August 2011 timing of the data collection for the UDB. 17 

Although the Suseti sample is not statistically representative of the whole country (or even the given 
districts), it has several unique features that make our results internally valid in terms of our primary goal 
of evaluating and decomposing the targeting performance of the UDB. First, the survey incorporated a 
rigorous matching process to enable identification of households registered in the UDB. We conducted 
desk-based matching using the names and addresses of household heads and spouses and also verified 
the matching results in the field. 18 The field-based verification process makes 'false positive' matches 
very unlikely, however, a small number of 'false negative' matches may occur (i.e., Suseti households 



14 The survey initially included 5,998 households, but there was attrition of about 5% (or 316) original households between 
the baseline and end line waves. We focus in the paper on the 5,682 households surveyed in both waves. Attritors do not 
systematically differ from non-attritors along baseline characteristics used in Suseti and in the UDB to construct the PMT. 
Results available upon request. 

15 For more detailed information on the design and sampling of the survey, which was originally collected to compare different 
targeting methods in a high-stakes experiment, see Alatas et al. (2013a and b). 

16 According to nationally representative household survey data from 2010 (Susenas), within the entire Indonesian population, 
about two-thirds of households have at least one child aged below 16 years old. 

17 An additional reason for using the baseline data is that the survey was administered before the government conducted any 
socialisation or targeting for PKH, while the end line was administered after the PKH programme started. Thus, we avoid using 
the consumption data in the end line, as it may potentially reflect nonrandom shocks associated with the PKH programme. 

18 Before the Suseti end-line survey was conducted, a listing of all households to be surveyed was constructed based on 
baseline respondents. This list was electronically matched with the UDB using household characteristics such as the addresses 
and names of the household head and spouse. This list was also matched with the enumeration pre-listing in order to identify 
households that were initially on this list but were not registered in the UDB. During the end-line survey fielding, enumerators 
and community leaders were asked to verify that the electronic matches were correctly identified. They were also asked to 
identify manually any other matches not yet identified by comparing the Suseti listing with the UDB registry. 

6 



who were also in the UDB but the match was not detected) due to the difficulty in recognising different 
versions of names. The expected effects of such potential undermatching would be to inflate slightly the 
estimated errors of exclusion and to deflate slightly estimated errors of inclusion. 

Our evaluation of the UDB's targeting performance is therefore based on comparing actual administrative 
data on household eligibility status for government social programmes (from the UDB) and data on 
their expenditures (from Suseti). This feature allows a better understanding of what happens at the 
multiple stages of the targeting process before benefits are delivered to households, including the 
decision on which potentially poor households should be surveyed, as well as the process of estimating 
their socioeconomic status based on the data collected. 19 

A second important feature of Suseti is the availability of information on receipt of Indonesia's main 
social protection programmes (Raskin, Jamkesmas, and BLT) before establishment of the UDB. These 
programmes relied in the past on different methods of identifying beneficiaries, such as using previous 
censuses of the poor (BLT) and/or nominations from community leaders. This allows us to compare the 
performance of the centralized UDB targeting registry, with more ad hoc (baseline) targeting approaches. 
We are therefore able to evaluate the change in targeting accuracy for programmes transitioning to using 
the UDB. 20 

Suseti also includes all the indicators used to calculate households' PMT scores in the UDB. This allows 
simulating the PMT process used in Indonesia under the hypothetical scenario of all households having 
been surveyed for inclusion in the UDB, rather than only the subset of households expected to be poor. 
We are thus able to distinguish between targeting errors that are due to 'enumeration errors', that is, 
poor households not registered in the UDB and those associated with the PMT estimation process. 

Furthermore, Suseti contains other types of information relevant to identifying determinants of targeting 
errors, such as household participation in the community, difficult-to-observe assets, and exposure to 
shocks. The survey component administered to the head of each hamlet/neighbourhood also collects 
information on community-level characteristics such as its geographic remoteness, the mode of selection 
of its head, and his/her social networks with other community members. 21 

Table 1 shows the results of the matching process used to determine which households from the Suseti 
survey are registered in the UDB. Overall, 41% of the PKH eligible population in the Suseti districts 
is registered in the UDB. In Suseti, of the 5,682 households surveyed, 2,444 or 43% are registered 
in the UDB. 22 Given regional variation in rates of poverty and vulnerability, this percentage differs 



19 The targeting accuracy of social programmes is measured based on a comparison of the discrepancy between intended 
and actual recipients, that is, who is poor (often based on household expenditures) and who is receiving government benefits. 
Existing evaluations of targeting accuracy (e.g., Coady et al. 2004) are usually done using data on both of these key indicators 
from a single survey. As a result, these evaluations rely on households to self-report whether they receive benefits rather than 
using administrative data directly. 

20 At the time of the fielding of Suseti (and matching with the UDB) in early 20 1 2, the UDB had not yet been used for targeting 
purposes. However, it was known which households were to be included in the beneficiary lists from the UDB provided to 
these programmes, based on their PMT score rankings. 

21 RTs are neighbourhood associations with all households registered as living in the area as members. By law, RTs are meant 
'to help smooth the execution of duty in administration, development, and social activities at the village and town level' 
(Kurasawa 2009). 

22 Suseti was also matched with the enumeration pre-listings; an additional 1,048 households that were removed from these 
pre- listings and are therefore not registered in the UDB were identified. 



across districts, from 28% of the Suseti sample in Wonogiri to 52% in Central Lampung and Pemalang. 
Overall, however, the high correlation in the shares of the population registered in the UDB in the Suseti 
sample and in the total population increases confidence in the accuracy of the matching exercise, which 
is important to ensuring valid estimates of targeting errors in the UDB. 23 



Table 1. Results of Dataset Matching: Share of UDB Households in the Total Population and in Suseti 





Total Population 








Suseti Sample 




District 


UDB 


All 


Share in 
UDB 

(%) 


UDB 


All 


Share in 
UDB 

(%) 


Bandar Lampung 


81,003 


223,730 


36 


215 


459 


47 


Central Lampung 


58,576 


132,554 


44 


739 


1408 


52 


Ogan Komering Ilir 


82,110 


226,705 


36 


344 


1056 


33 


Palembang 


53,693 


149,010 


36 


380 


826 


46 


Pemalang 


121,031 


211,100 


57 


490 


949 


52 


Wonogiri 


50,040 


138,369 


36 


276 


984 


28 


All 


446,453 


1,081,468 


41 


2,444 


5,682 


43 



Notes: This table shows the percentage of PKH-eligible households, that is, households with children aged under 16 years old, registered in 
the UDB, comparing UDB/Susenas data with Suseti data for the six sample districts. The first group of columns shows the total number of 
households with children aged under 16 years old recorded in the UDB ('UDB' column) and in the full population from Susenas 2010 data 
('All' column). The second group of columns shows the number of households from the Suseti data successfully matched with the UDB 
administrative data ('UDB') and the total number of households in Suseti ('All'). 



3.2 Comparison of UDB-Registered and Nonregistered Households 

Table 2 provides an initial glimpse into the UDB's performance in reaching the poorest households, 
through a comparison of the socioeconomic characteristics of Suseti households registered in the 
UDB and those not included ('non-UDB'). Households registered in the UDB appear significantly 
poorer, with monthly per capita expenditure levels 1 .4 times lower on average than those of non-UDB 
households. Compared with non-UDB households, UDB households tend to have significantly more 
family members and children. UDB household heads also have about two fewer years of schooling, and 
fewer among them are male and working, compared with non-UDB household heads. 

Table 2 also shows that UDB households are more likely than non-UDB households to have previously 
received benefits from one or more of the national social protection programmes distributed before 
implementation of the UDB. 24 For the BLT cash transfer programme distributed in 2008, 58% of UDB 
households reported having been recipients compared with 26% of non-UDB households; the figures 



23 Matching rates in the urban districts of Bandar Lampung and Palembang appear relatively higher than the share of the 
population registered in the UDB, suggesting that local-level characteristics may affect the matching rate. However, similar 
results to table 1 are obtained when considering district-specific average village shares of UDB households in the population 
and in the Suseti sample. 

24 Note that because the Suseti data were collected before any of these social programmes had begun to use the UDB for 
selecting beneficiaries, these baseline figures indicate numbers of previous beneficiaries entering into the UDB and do not 
show the UDB's anticipated effects on programme targeting outcomes, which are explored later. 



are similar for the Jamkesmas health fee waiver programme (59% and 33%, respectively). For the 
Raskin subsidised rice programme, 92% of UDB households reported having received programme 
benefits compared with 71% of non-UDB households. 25 



Table 2. Socioeconomic Characteristics of Households in the Suseti 





All 

Households 


UDB 


non 
UDB 


t-stat 


Demographic Characteristics 


Household size 


4.8 


4.9 


4.6 


-6.03*** 


Number of Children aged 0-15 years 


1.7 


1.8 


1.6 


-7.36*** 


Household head aged 


44.4 


44.1 


44.5 


1.28 


Male household head 


0.95 


0.93 


0.95 


2.50** 


Household head schooling years 


6.9 


5.9 


7.7 


16.84*** 


Household head works 


0.93 


0.92 


0.94 


2.15** 


Household head works in agricultural sector 


0.45 


0.46 


0.44 


-1.79* 


(Baseline) household expenditures per capita, IDR 


575,766 


471,443 


654,499 


16.54*** 


Receipt of Social Assistance Programmes 


Raskin subsidised rice 


0.80 


0.92 


0.71 


—19 59*** 


Jamkesmas health waiver programme 


0.44 


0.59 


0.33 


-20.19*** 


BLT unconditional cash transfer in 2008 


0.40 


0.58 


0.26 


-25.44*** 



Notes: This table reports averages for all households in Suseti followed by a breakdown for households in the UDB and not in the UDB. 
Cells with values less than one are variables reporting a proportion. Per capita expenditures are nominal rupiah values as reported in the 
baseline survey. The t-stat is based on a two-sided test for difference in means between UDB and non-UDB households. Stars indicate 
significance at the 1% ***, 5% **, and 10% * level. 



The finding that UDB households are poorer than non-UDB households is further supported in figures 
la and lb, which indicate the targeting performance of UDB. Figure la confirms the findings of table 2 
in showing that UDB households are on average poorer than non-UDB households are. However, there 
appears to be a rather large overlap in the consumption distributions, suggesting that a sizable share of 
poor households is not in the UDB. 26 Figure lb plots the probability of being in the UDB against per 
capita consumption and shows a clear inverse relationship. Households with the lowest consumption 
levels have a probability of being in the UDB of more than 60%, whereas this probability is lower than 
20% for households with the highest consumption levels. In theory, these probabilities would ideally 
be 100% and 0%, respectively. However 'perfect targeting' performance is impossible in practice. The 
next section explores how the expected UDB targeting outcomes fare regarding baseline benchmarks 
for the three major social assistance programmes. 



25 It is notable that the average numbers of households reporting receiving benefits from each programme appear to be quite 
high, given that each of these programmes is intended to cover about 20% to 30% of households on average nationally. This 
could reflect either specificities of the Suseti sample or the dilution of programme benefits. For Raskin, for instance, there is 
evidence that the fixed allocations of 1 5 kilograms of subsidised rice normally targeted to poor households are often distributed 
more widely, including among the entire community (SMERU 2008). World Bank (2012) also finds using the nationally 
representative Susenas, that 50%, 30%, and 27% of households report receiving respectively Raskin, Jamkesmas, and BLT. 

26 Note that there may be measurement error affecting the spread of the consumption distribution. 



Figure 1A. Distribution of per Capita Expenditures UDB and Non-UDB Households 

in the Suseti Sample 
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Figure IB. Probability of being in the UDB and Actual per Capita Expenditures 
1 - 



.8 - 




12 13 14 15 16 

Log per capita expenditures 



Notes: Figure 1(a) plots the kernel density of log household expenditures per capita separately for Suseti households registered and not 
registered in the UDB. Figure 1(b) plots the local linear probability of being in the UDB against log expenditures. 
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4. Evaluating the Targeting Performance of the UDB 



This section analyses in greater depth the targeting performance of the UDB, taking advantage of 
the matched Suseti-UDB data. Section 4.1 explains the methodology used to assess baseline and 
expected targeting accuracy of social programmes in Indonesia. Section 4.2 presents the UDB's 
targeting performance. Section 4.3 takes the analysis further by disentangling errors resulting from the 
enumeration process and PMT classification errors. 

4.1 Methodology for Assessing the Targeting Accuracy of the UDB 

A large literature examines different measures and methodologies for estimating targeting accuracy 
(see Coady et al. 2004 for a review). Commonly used measures of targeting outcomes include 
undercoverage and leakage (Cornia and Stewart 1995), the distributional characteristic (Coady and 
Skoufias 2004), and the Coady-Grosh-Hoddinott measure (Coady et al. 2004). In this paper and in 
line with most of the literature, we use undercoverage and leakage as our main measures of targeting 
outcomes. Undercoverage — or exclusion error — is defined as the share of households below a given 
poverty threshold that are not receiving programme benefits. We consider two thresholds specifically and 
define undercoverage using the 30th percentile of household actual (adjusted) per capita consumption 
in the Suseti sample, and severe undercoverage using the 10th percentile. 27 Conversely, leakage, or 
inclusion error, is defined as the share of households that are above a given threshold and yet receive 
benefits. Similar to undercoverage, we use two thresholds and define leakage using the 60th percentile 
and severe leakage using the 80th percentile of the adjusted per capita consumption distribution. Key 
results are robust to alternative thresholds. 

As described earlier, we assess the targeting performance of the UDB against the baseline targeting 
performance of the main social assistance programmes (Raskin, Jamkesmas, and BLT). More specifically, 
we consider the performance expected from the use of lists of eligible beneficiaries from the UDB. 
Focusing on pre- determined eligibility based on the UDB, rather than on reported receipt of benefits, 
allows us to emphasize the potential for the newly established UDB to improve targeting outcomes, 
setting aside other programme implementation issues that might affect benefit delivery. However, 
some discrepancy between the expected and actual UDB targeting errors might occur depending on 
the degree of compliance with the beneficiary lists extracted from the UDB in the field. For Jamkesmas 
and BLT, the amount of discrepancy is expected to be relatively small because eligibility cards were 
printed directly based on the UDB. For Raskin, more discrepancy is anticipated between expected and 
actual targeting outcomes due to the longstanding community practice of sharing rice benefits across 
nominally eligible and ineligible households. 28 

Baseline targeting errors are calculated by comparing reported programme receipt to household per 
capita consumption. Expected UDB targeting errors are calculated for households registered in the 
UDB by comparing actual per capita consumption (from Suseti) with the PMT scores (from the UDB) 



27 These levels correspond closely to the thresholds used by the Indonesian government to determine eligibility for its main 
social assistance programmes; BLT, Jamkesmas, and Raskin all cover roughly the poorest 30% of households in the country, 
while the eligibility threshold for PKH is close to the poorest 10%. 

28 One of the objectives of establishing the UDB is to reduce benefit dilution, which decreases significantly the share of 
benefits received by the target population. Raskin beneficiaries, for instance, purchased on average 3.8 kilograms of rice 
monthly, instead of the intended 15 kilograms, 'due to community-level sharing of benefits by non-target households' (World 
Bank 2012). 



used to produce beneficiary lists based on each programme's eligibility threshold. Any household in the 
Suseti sample not found in the UDB through the matching process is considered a non-recipient. 

The comparison of baseline and expected UDB targeting errors (undercoverage and leakage) reveals 
the change in targeting performance due to the transition of the programmes to using the UDB for 
beneficiary selection. Switching to the UDB implies changes not only in which households will receive 
programme benefits but also in the total number of beneficiaries (i.e., programme coverage). These 
differences in coverage make it difficult to compare across time and across programmes. Therefore, 
we first present standard undercoverage and leakage measures to assess the overall change in targeting 
performance before (baseline) and after the UDB. We then isolate the change expected solely from 
beneficiary identification using the UDB lists by computing UDB undercoverage and leakage at an 
unchanged (baseline) coverage level. 

We also address a notable limitation of standard undercoverage and leakage measures (see, for example, 
Coady and Skoufias 2004 and Coady et al. 2004), which weight all households equally, regardless 
of their position in the consumption distribution. For instance, when measuring undercoverage for a 
programme intended to cover the poorest three deciles of the consumption distribution, no distinction 
is made between the exclusion of a household in the poorest 5% and that of a household in the 29th 
percentile, even though from a welfare perspective, excluding the former represents a more serious 
error. We therefore also present the expected incidence of benefits across all consumption deciles to 
provide a more detailed assessment of the distributional performance of the UDB. 

4.2 Results: UDB Targeting Performance 

In this section, we evaluate the overall targeting performance of the UDB through the changes in 
targeting accuracy that can be expected from the transition of the three main Indonesian social assistance 
programmes to using the UDB. Column 1 of table 3 shows that, at baseline, 80%, 44%, and 39% of 
all Suseti households report to have previously received, respectively, Raskin, Jamkesmas, and BLT 
Compared with Jamkesmas and BLT, Raskin's substantially higher coverage levels have led to (1) very 
low baseline undercoverage: less than 11% of the poorest three deciles have not received the subsidised 
rice benefits and (2) high leakage: 74% of the richest four deciles have received benefits. Jamkesmas 
and BLT have similar baseline targeting errors with, respectively, leakage rates of 34% and 39% and 
undercoverage rates of 45% and 51%. These patterns in targeting errors align with previous research 
analysing the targeting performance of Indonesia's social protection programmes before establishment 
of the UDB (World Bank 2012). 

Results presented in column 2 of table 3 show first that Raskin and Jamkesmas coverage levels decrease 
significantly with the UDB compared with baseline. This reduction in the number of beneficiaries 
automatically leads to an increase in undercoverage for both Raskin (from 1 1 % to 54%) and Jamkesmas 
(from 45% to 53%). Another consequence of the expected decrease in coverage is that leakage to non- 
poor households is expected to decrease significantly with Raskin and Jamkesmas using the UDB. 
These improvements are most apparent for Raskin, where the baseline leakage of 74% is expected to 
decrease by 50 percentage points with the UDB. For Jamkesmas, baseline leakage rates are expected 
to fall from 39% to 25% with the use of the UDB, with severe leakage falling even further from 33% 
to 17%. 
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Similar patterns are observed when focusing on the 'severe' measures of undercoverage and leakage 
listed in table 3, which are lower across all programmes, at baseline, and with the UDB. 



The difference between baseline and expected UDB errors is difficult to interpret, given the substantial 
change in programme coverage levels associated with the transition to using the UDB for selecting 
beneficiaries. It is therefore useful to keep coverage levels constant as an alternate way to assess the 
change in targeting performance expected from programmes that transition to using the UDB. We do 
this using baseline BLT 2008 coverage levels and identify households that would be eligible for a 
programme with such coverage based on the UDB. We choose to match BLT 2008 coverage levels 
because previous research (World Bank 2012) indicates that the BLT 2008 has the highest targeting 
accuracy among the three social programmes considered, and thus using it as a benchmark provides the 
strictest possible test of the UDB's performance relative to baseline. 29 



Table 3. Baseline and Expected UDB Programme Targeting Accuracy 



Targeting Measures 


(1) 


(2) 


Baseline (%) 


Expected UDB (%) 


Panel A: Raskin 


Coverage level 


80 


31 


Leakage 


74.4 


23.5 


Severe leakage 


66.7 


16.4 


Undercoverage 


10.9 


54.0 


Severe undercoverage 


7.7 


49.6 


Panel B: Jamkesmas 


Coverage level 


44 


33 


Leakage 


38.7 


24.9 


Severe leakage 


32.5 


17.4 


Undercoverage 


44.9 


52.6 


Severe undercoverage 


42.8 


48.4 


Panel C: BLT 2008 


Coverage level 


39 


39 


Leakage 


34.2 


31.5 


Severe leakage 


27.0 


24.0 


Undercoverage 


50.5 


47.7 


Severe undercoverage 


48.1 


44.2 



Notes: This table reports estimates of targeting errors, computed separately for 2008 Raskin, Jamkesmas, and BLT programmes. Leakage 
(severe leakage) captures the fraction of households with adjusted expenditures per capita at baseline above the 60th (20th) percentile of 
the consumption distribution that receive the given programme. Similarly, undercoverage (severe undercoverage) captures the fraction of 
the poorest 30% (10%) that do not receive the given programme. The definition of programme receipt varies across columns. In column 
1, programme receipt is as reported by households in Suseti. In column 2, programme receipt equals one if the household's PMT score in 
the UDB places it within the pool of intended programme recipients. In this column, the BLT 2008 programme is based on ranking the 
household PMT scores and taking all households with PMT scores up to the number of households reporting BLT 2008 receipt in Suseti. 
This calculation is done within each of the six districts in keeping with the district-specific targeting quotas. 



29 In addition, only 43% of Suseti households are matched with the UDB, which is lower than Raskin and Jamkesmas baseline 
coverage levels at respectively 80% and 44%. 
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As shown in column 2, panel C, of table 3, holding coverage levels constant, using the UDB would 
reduce undercoverage and severe undercoverage of the BLT programme from 51% to 48% and from 
48% to 44%, respectively. This decrease in exclusion errors further suggests that the main reason for 
the increase in undercoverage expected with usage of the UDB for Raskin and Jamkesmas noted earlier 
in this section is the concurrent decrease in their coverage levels, compared with baseline. Leakage 
also decreases for this simulated BLT programme from 34% to 32%. To summarise, holding (BLT 
2008) coverage levels constant, the UDB is predicted to improve both undercoverage and leakage, 
representing a roughly 6% improvement in each of the respective errors relative to baseline. 

As to distribution of programme receipt across household per capita consumption deciles, figure 2 shows 
that targeting for all programmes is rather progressive. At both baseline and with the UDB, a larger share of 
households from the poorest consumption deciles is receiving each programme, compared with households 
from the richest consumption deciles. The graphs confirm that the UDB has led to an improvement in the 
targeting performance of the three main Indonesian social assistance programmes. There is a considerable 
reduction in leakage with the UDB compared with baseline. The probability of receiving benefits decreases 
faster as per capita expenditures increase with the UDB, compared with baseline for all programmes, 
despite the large decrease in coverage observed for Raskin and, to a lesser extent, for Jamkesmas. At a 
constant coverage level (for BLT), the difference is lower but remains significant. The difference in the 
slopes of the lines predicting the probability of receiving any of the three programmes at baseline and with 
the UDB is positive and significant. 30 This implies that targeting using the UDB is more progressive than 
with the previous approaches to beneficiary selection used in Indonesia. 



Figure 2. Programme Benefit Incidence Baseline and UDB 

Raskin Jamkesmas BLT 

1 ■ 




12 12.5 13 13.5 14 14.5 15 12 12.5 13 13.5 14 14.5 15 12 12.5 13 13.5 14 14.5 15 

Log per capita expenditures Log per capita expenditures Log per capita expenditures 

Baseline UDB 

Notes: This figure shows the probability of receiving each programme at baseline and with the UDB as a function of adjusted per capita 
expenditures, estimated using local linear regressions. Baseline programme receipt and per capita expenditures are from Suseti. UDB pro- 
gramme receipt is based on beneficiary lists from the UDB. For BLT, UDB programme receipt is based on ranking household PMT scores 
and taking all households with PMT scores up to the number of households reporting BLT 2008 receipt in Suseti. 



30 This is confirmed by the results of a Wald test, available upon request. 
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4.3 Disentangling Misenumeration and Misclassification 



As described earlier, targeting errors in the UDB can be attributed to two factors: (1) misenumeration, 
or undercoverage of poor households during the enumeration process and (2) misclassification of 
households during the PMT modelling stage. This section attempts to disentangle these two sources of 
errors using 'reconstructed' PMT scores calculated for all households in the Suseti sample, instead of 
focusing only on those matched households that are actually registered in the current UDB. This permits 
assessment of UDB performance that would be observed if all households had been registered and 
scored in the UDB, rather than only surveying households expected to be poor (based on the pre-listings 
from poverty mapping and consultation with community members). Simulating outcomes under this 
census scenario makes it possible to remove errors, as poor households are not enumerated, and instead 
to isolate the role of the PMT process in contributing to targeting errors. 

We reconstruct PMT scores for all households in the Suseti sample by applying the PMT algorithms 
used by UDB planners to the underlying PMT variables collected from each household in Suseti. We 
then calculate targeting errors by comparing programme eligibility status (based on the reconstructed 
PMT scores and on UDB-based coverage levels; see column 2, table 3) against household expenditure 
rankings (from Suseti). 

Table 4 shows the improvement in targeting errors expected under this full census scenario relative 
to the UDB targeting errors presented earlier. More specifically, the measures presented in table 4 are 
computed as the difference between the targeting errors presented in columns 1 and 2 of table 3 and 
the ones obtained when assigning programme receipt to all households with reconstructed PMT scores 
below the given programme eligibility threshold, as a share of the UDB errors from table 3 : 

rP _ U PMT UDB ~ U PMT SST 
UPMT UDB 

Where U refers to the different measures of undercoverage and leakage used in the previous section; 
PMT UDB refers to household eligibility status based on actual PMT scores from the UDB; and PMT SST 
refers to eligibility status based on PMT scores reconstructed using the underlying PMT variables from 
SUSETI. 



A negative (positive) sign indicates a decrease (increase) in targeting errors if all households had been 
registered in the UDB. Both leakage and undercoverage rates in the UDB are projected to improve under 
this scenario across all programmes by 11%-17% and 13%-17%, respectively. The improvements are 
even more striking for severe leakage and particularly severe undercoverage as gains in the latter range 
27%-36% across programmes. In other words, it appears that expanding the number of households 
enumerated in the national targeting survey holds significant potential for improving targeting outcomes, 
particularly by reducing exclusion of the poor. 

Figure 3 includes the predicted probability of receiving programme benefits based on the PMT scores 
reconstructed for all households using the data collected in Suseti in the comparison of benefit incidence 
between baseline and UDB-based (for households registered in the UDB) programme receipt presented 
in figure 2. In line with results from table 4, households from the poorest three consumption deciles 
have a higher probability of receiving programme benefits when considering PMT-specific predictions 
for all Suseti households, as opposed to UDB households only. The census scenario (reconstructed 
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PMT scores) also leads to some improvements in leakage, as shown by the lower programme receipt 
probability for households from the richest half of the population. 

We use the results of this exercise to demonstrate the cost-effectiveness of expanding enumeration to 
cover the full population. It is estimated that, under a census scenario, households from the poorest three 
deciles would be more likely to receive Raskin and Jamkesmas by about 6 percentage points and more 
likely to receive BLT by about 8 percentage points. From this, we extrapolate that an additional 1 . 1 million 
households from the poorest three deciles would receive benefits from these three programmes under a 
full enumeration scenario. Benefit levels of Raskin, Jamkesmas, and BLT amount annually to a total of Rp 
2.4 million (about US$200) per household or Rp 1.2 million, 0.96 million, and 0.2 million respectively. 31 
Assuming that the unit cost of surveying the remaining 60% of the population would be similar to the one 
incurred in establishing the UDB 32 — about Rp 25,000 (US$2) per household for a survey done during 
a three-year period — it follows that surveying the full population would cost about 11% of the value of 
additional benefits that would be received annually by households from the poorest three deciles. 



Table 4. Expected Improvement in Targeting Accuracy with Full Census Enumeration 



Targeting Metrics 


Expected Change in Targeting Error, Compared to UDB (%) 


Panel A: Raskin 


Leakage 


-16.6 


Severe leakage 


-19.5 


Undercoverage 


-13.0 


Severe undercoverage 


-27.2 


Panel B: Jamkesmas 


Leakage 


-16.1 


Severe leakage 


-18.4 


Undercoverage 


-14.1 


Severe undercoverage 


-30.8 


Panel C: BLT 2008 


Leakage 


-11.4 


Severe leakage 


-18.8 


Undercoverage 


-16.8 


Severe undercoverage 


-35.5 



Notes: This table reports estimates of UDB targeting errors based on reconstructed PMT scores for all Suseti households (i.e., simulating 
a scenario of full census enumeration). Programme receipt equals one for all households with reconstructed PMT score rankings that 
fall below the number of households eligible to receive programmes based on the UDB. The BLT 2008 eligibility is based on the same 
procedure as in table 3, that is, ranking the household PMT scores and taking all households with PMT scores up to the number of 
households reporting BLT 2008 receipt in Suseti. 



31 Raskin benefits are valued at Rp 90,000 per household per month, based on provision of 1 5 kilograms of rice per month at a 
subsidy equivalent to about Rp 6,000 per kilogram compared to the market price. For Jamkesmas, the value of the premium of 
the newly established national health insurance, which is currently paid for by the government for households from the poorest 
40%, is used; this amounts to about Rp 80,000 per month for a household of four members. For BLT, the value of benefits 
provided in 2013 — Rp 600,000 per household per year — is divided by three, assuming that this temporary compensation 
programme is only implemented once during a three-year period. In the past, BLT provided households with Rp 1.2 million 
(2005) and Rp 900,000 (2008). 

32 Based on the 2010 Population Census, there are a total of about 61.2 million households. 



Figure 3. Programme Benefit Incidence Baseline, UDB and Reconstructed PMT Scores 
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Notes: Baseline programme receipt and per capita expenditures are from Suseti. 'PMT in UDB' refers to Suseti households matched with 
the UDB and ranked using their PMT scores from the UDB. 'Reconstructed PMT' refers to all households in the Suseti sample ranked ac- 
cording to their PMT score reconstructed using the underlying PMT variables from Suseti. Similar results are obtained when also using the 
reconstructed PMT scores for households matched with the UDB. 



5. Towards Explaining Targeting Performance 



This section explores individual- and village-level factors that are associated with misenumeration 33 
and PMT misclassification in order to gain deeper insight into targeting processes and effectiveness. 
First, as noted earlier, during the enumeration process, households have been removed and added to 
the pre-listing of households to be surveyed, which was constructed using the 2010 Population Census 
and Elbers et al. (2003) poverty mapping methodology Misenumeration may occur as a result of the 
addition and removal of households if households added (removed) have on average a higher (lower) 
socioeconomic status than those that end up being registered in the UDB. Second, during the PMT 
modelling process, a limited number of observable household characteristics were used to predict 
welfare levels. Misclassification may occur if these observable characteristics only capture a limited 
share of the relevant overall variation in household welfare. 

5.1 Unpacking the Enumeration Process 

This section considers which of the different methods used for registering households in the UDB led to 
surveying poorer households. In addition to the initial roster of households to be surveyed (pre-listing), 
community suggestions were used to identify additional poor households during the data collection. 
We also match Suseti with the enumeration pre-listings and identify 1,048 households removed from 
these pre-listings because they were considered to be rich and therefore not registered in the UDB. 
Figure 4 shows that the consumption distribution of UDB households surveyed with the pre-listing is 
slightly more to the left (poorer) compared with that of UDB households identified through community 
suggestions ('in UDB, not on pre-listing'), and both of these distributions are poorer compared with 
households not in the UDB, in line with figures la and lb. Households that had been removed from the 
survey enumeration pre-listing and therefore not registered in the UDB have a consumption distribution 
similar to other households not registered in the UDB. 

Figure 4. Distribution of per Capita Expenditures, by Registration Channel 
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33 Note that the term 'misenumeration' here comprises all possible implementation errors that occurred during the process of 
registering households in the UDB. We are not able to distinguish fully between errors caused by the poverty map and those 
caused by suggestions from local communities. 
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To analyse additional factors that predict household registration in the UDB or having been removed 
from the enumeration pre-listing for being considered rich (see section 2. 1), we estimate the following 
Probit models: 

Pr(UDB h = 1 | PCE h ,X h ,Z vh ) = 0(a x + ftln(PCB fc ) + X hYl + Z vh S x ) (2) 

Vr{Removed h = 1| PCE h ,X h ,Z vh ) = <t>(a 2 + p 2 \n(PCE h ) + X h y 2 + Z vh S 2 ), (3) 

Where UDB h is equal to one if household h is registered in the UDB, zero otherwise; and Removed h is 
equal to one if household h has been removed from the enumeration pre-listing, zero otherwise. 

The set of variables X and Z are selected to reflect household- and community-level characteristics 
that are not included in the process of determining enumeration quotas or the calculation of PMT 
scores but may affect misenumeration through their correlation with household welfare and/or local 
implementation features. 34 At the household level, we consider several factors associated with household 
'hidden assets' 35 as relevant to the enumeration process: exposure to shocks, and social connectedness 
and position within the community. Controlling for expenditure levels, we expect households owning 
hidden assets to be less likely to be registered in the UDB and more likely to have been removed from 
the enumeration pre-listing. On the contrary, -households that experience shocks, have more social 
connections, and are considered poor within their community should be more likely to be registered 
in the UDB. At the community level, we consider indicators of the relative economic status of the 
community, as well as the potential for elite capture, proxied by the characteristics of community 
(neighbourhood) heads and the remoteness of the village. We expect households living in relatively 
poorer communities that are less vulnerable to elite capture to be more likely to be registered in the 
UDB. It should be emphasised that these regressions are merely conditional correlations, and we do not 
intend to assign a causal interpretation. 

Results for equation (2) are presented in columns 1 and 2 in table 5, whereas results for equation 
(3) are presented in columns 3 and 4 in table 5. The negative and significant coefficient associated 
with household per capita consumption aligns with the findings of figure lb. Interestingly, per capita 
consumption has no significant correlation with the probability of removal from the enumeration pre- 
listing because of being 'rich'. This is consistent with recent evidence suggesting that the definition of 
being poor used by communities may only partially be correlated with household per capita consumption 
(Alatas et al. 2012). Alternatively, it could indicate a certain degree of elite capture over the process of 
determining which households are registered in the UDB. 

Conditional on their level of consumption, households that own partially hidden assets, such as land 
or gold, are less likely to be registered in the UDB and more likely to be removed from the pre-list, 
suggesting that communities may not have abused this possibility to remove households. This is 
consistent with the argument that local communities have better information on the socioeconomic 
status of their members (Dreze and Sen 1989). 



34 Table A 1 in annex lists all variables and their summary statistics. 

35 The term 'hidden assets' is used to refer to assets that are difficult for enumerators to observe directly and therefore more 
subject to misreporting. It is commonly advocated to avoid using such easily manipulable indicators for estimating PMT 
scores, due to the increased probability of misreporting, especially when respondents are aware that the survey is being 
conducted for the purpose of selecting beneficiaries for social assistance programmes. In Colombia, Camacho and Conover 
(2011) provide evidence that, when PMT formulas become known, there is an increase in misreporting to increase one's 
chances of receiving programme benefits. 



Controlling for socioeconomic status, several proxies for social connectedness are associated with a 
higher probability of registration in the UDB. For example, migration is associated with a higher chance 
of registration in the UDB; two likely explanations are that (1) migration is a way for households to cope 
with economic hardship and hence these households are relatively poor and (2) migrants must register 
with the village head, suggesting that these households and those that have family connections in the 
neighbourhood (which also increase the probability of being in the UDB) are known to community 
leaders and therefore less likely to be 'missed' during the enumeration process, conditional on their 
expenditure levels. Meanwhile, proxies for household position within the community, in particular the 
receipt of assistance from the community, are also associated with a higher probability of registration in 
the UDB. Interestingly, receiving nongovernmental assistance is associated with a higher probability of 
removal from the enumeration pre-listing and also of ultimate inclusion in the UDB. Receipt of zakat 36 , 
however, is associated with a higher probability of registration in the UDB and a lower probability of 
removal from the pre-listing. Again, this suggests the coexistence of different definitions of poverty 
used by communities (Alatas et al. 2012). There may also be a concern for fairness within communities; 
as a result, households that have access to alternative forms of support when facing hardship may be 
more likely to be removed from the pre-listing. 

At the community level, having an RT head who considers the district to be poorer than other districts 
is associated with a higher probability of enumeration. This aligns with findings of SMERU (2012) 
indicating that community leaders are reluctant to survey a high number of households and particularly 
those not considered to be poor, as this would raise households' expectations on receiving programme 
benefits. Among indicators of the potential for elite capture, households in communities where the RT 
head declares that s/he knows each community member very well have a lower probability of removal 
from the enumeration pre-listing. Village remoteness is negatively associated with registration in the 
UDB and positively with removal from enumeration pre-listings, all else being equal. These areas 
may be more difficult or more costly for enumerators to reach, and supervision of enumerators may be 
lacking in these areas, potentially leading to local leaders that have more leeway in removing people 
from the pre-listing (even if they deserve to be registered in the UDB). 



36 The reception of zakat is also included in the nongovernmental assistance dummy (and in the annual per capita value), 
which also comprises assistance received from religious or political institutions, as well as from national and international 
nongovernmental organisations, firms/corporations, and other private donors. Assistance received in response to disaster is 
excluded. 
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Table 5. Hidden Household- and Village-Level Characteristics Associated with Registration in the UDB 







Pr 




Pr 


Variables 


(in the UDB) 


(Household removed from PL a ) 




1 


2 


3 


4 


Log per capita expenditures 


-0.512*** 
(0.051) 


-0.501*** 
(0.052) 


-0.029 
(0.056) 


-0.019 
(0.058) 


Asset: Land 


-0.162*** 
(0.061) 


-0.173*** 
(0.058) 


0.277*** 
(0.075) 


0.264*** 
(0.077) 


Asset: Jewellery or gold/savings > Rp 


-0.346*** 


-0.334*** 


0.117** 


0.124** 


500,000 


(0.038) 


(0.037) 


(0.050) 


(0.049) 


-A.ssei. LivesiocK wiin vaiue ^ Kp juu,uuu 


-0.028 
(0.055) 


-0.047 
(0.054) 


0.272*** 
(0.051) 


0.249*** 
(0.052) 


Household has experienced a shock in past 


0.046 


0.031 


0.070* 


0.066 


year 


(0.048) 


(0.047) 


(0.038) 


(0.041) 


Nb of family members living in the same 


0.022** 


0.016* 


0.012 


0.009 


RW/RT 


(0.010) 


(0.010) 


(0.010) 


(0.011) 


At least one household member has migrated 


0.085** 
(0.042) 


0.095** 
(0.040) 


0.030 
(0.040) 


0.037 
(0.040) 


At least one household member participates 


0.024 


0.035 


0.009 


0.004 


in community group 


(0.047) 


(0.050) 


(0.072) 


(0.070) 


Assistance received from nongovernmental 


0.473* 


0.384 


0.784** 


0.720** 


institution in past year 


(0.256) 


(0.253) 


(0.335) 


(0.340) 


Xcikcit was received in past year 


0.435*** 
(0.044) 


0.462*** 
(0.041) 


-0.214*** 
(0.061) 


-0.218*** 
(0.061) 


Log per capita value of total 


-0.042 


-0.032 


-0.087** 


-0.077** 


nongovernmental assistance, past year 


(0.028) 


(0.027) 


(0.035) 


(0.035) 


UDB enumeration quotas, percentage of 




0 921*** 




0.674*** 


village population 




(0.171) 




(0.163) 


Village is much or slightly poorer than other 




-0.014 




-0.083 


villages in district 




(0.079) 




(0.087) 


Village has similar income level as other 




-0.059 




-0.079 


villages in district 




(0.081) 




(0.083) 


District is much or slightly poorer than other 




0.092* 




-0.070 


districts 




(0.052) 




(0.043) 


(Log) distance between village and district 




-0.061 




-0.093* 


capital 




(0.052) 




(0.056) 


RT head knows each community member 




-0.087 




-0.009 


very well 




(0.061) 




(0.068) 


RT head is elected 




-0.110* 
(0.056) 




0.125** 
(0.059) 


Constant 


7.045*** 
(0.671) 


6.823*** 
(0.779) 


-1.098 
(0.763) 


-1.969** 
(0.808) 


Observations 


5,680 


5,680 


5,680 


5,680 


Pseudo R-squared 


0.113 


0.129 


0.0557 


0.0673 



' PL: pre-listing; RW: rukun warga (community association). 

Notes: This table reports the results of Probit estimates of the probability of registration in the UDB — columns 1 and 2 — and of being 
removed from the enumeration pre-listing for being considered rich - columns 3 and 4. All regressions include district fixed effects, and 
standard errors are clustered at the subdistrict level. Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. 
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5.2 Unpacking the PMT Classification 



This section examines the determinants of the gap between a household's actual per capita consumption 
and its PMT score, which indicates the extent of targeting errors due to PMT misclassification. In 
particular, we estimate the following two regressions: 

ln ( pJtudbJ = ai + X 'hYuDB + Z 'vh S UDB + 4ft ( 4 ) 

ln ( pmtjstJ = " 2 + X ' hY ssT + Z 'vhS 2 SS T + 4h (5) 

The more positive the dependent variable is for a household, the greater the extent that the PMT 
score underestimates actual consumption (i.e., potential inclusion error). Conversely, the more 
negative In(-^r) is, the greater the extent that the PMT score overestimates actual consumption (i.e., 
potential exclusion error). The difference between equations (4) and (5) is similar to the difference 
between the targeting error results presented in tables 3 and 4. Equation (4) estimates the correlation 
between misclassification and household and community characteristics conditional on actual registration 
in the UDB and uses household PMT scores from the UDB. By relaxing this sample restriction and 
considering all Suseti households based on their reconstructed PMT scores in equation (5), we are able 
to isolate the influence of the PMT process. Results for equation (4) are presented in columns 1 and 2 of 
table 6, and results for equation (5) are presented in columns 3 and 4 of table 6. 

Across all specifications, ownership of hidden assets of a value equal to or above Rp 500,000 (about 
US$40) is associated with underprediction of household consumption by the PMT formulas. All else 
being equal, actual expenditures of households owning such assets are between 6% and 14% higher on 
average than their estimated PMT scores. This is relatively unsurprising. However, use of these partially 
hidden assets is not recommended in PMT-based scoring, because they are not easily verifiable and 
therefore more likely to be manipulated by households. The occurrence of a shock at the household level 
in the past year is also significantly associated with potential leakage; actual per capita expenditures 
were on average 4% higher than PMT scores, conditional on registration in the UDB. This can be 
considered an 'acceptable' inclusion error, as shocks may render these households vulnerable to being 
poor. Moreover, receiving nongovernmental assistance and/or zakat is associated with having a PMT 
score or predicted welfare level between 10% and 50% higher than actual per capita expenditures. This 
further suggests that communities use a different definition of poverty. 
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Table 6. Hidden Household- and Village-Level Characteristics Associated with PMT Classification 

Errors 







OLS: 




OLS: 


Variables 


Log(PCE/PMT UDB) 


Log(PCE/PMT SST) 




1 


2 


3 


4 


Asset: Land 


0.033 
(0.028) 


0.028 
(0.027) 


0.013 
(0.018) 


0.014 
(0.018) 


Asset: Jewellery or gold/savings > Rp 500,000 


0.110*** 
(0.023) 


0.106*** 
(0.023) 


0.137*** 
(0.015) 


0.136*** 
(0.015) 


Asset: Livestock with value > Rp 500,000 


0.062** 
(0.028) 


0.058** 
(0.029) 


0.079*** 
(0.017) 


0.079*** 
(0.017) 


Household experienced a shock in past year 


0.044* 
(0.022) 


0.045** 
(0.022) 


0.019 
(0.017) 


0.019 
(0.016) 


Nb of family members living in the same RW7 


0.007 


0.006 


0.006** 


0.006** 


Kl 


(0.005) 


(0.005) 


(0.003) 


(0.003) 


At least one household member migrated 


0.023 
(0.020) 


0.025 
(0.020) 


0.004 
(0.013) 


0.004 
(0.012) 


At least one household member participates in 


0.038 


0.044 


0.032* 


0.034* 


community group 


(0.030) 


(0.030) 


(0.019) 


(0.019) 


Assistance received from nongovernmental 


-0.510*** 


-0.506*** 


-0.506*** 


-0.501*** 


institution in past year 


(0.130) 


(0.126) 


(0.074) 


(0.072) 


Zakat received in past year 


-0.107*** 
(0.026) 


-0.108*** 
(0.026) 


-0 114*** 
(0.016) 


_0.1 14*** 
(0.016) 


Log per capita value of total nongovernmental 


0.057*** 


0.058*** 


0.052*** 


0.051*** 


assistance, past year 


(0.015) 


(0.014) 


(0.008) 


(0.008) 


UDB enumeration quotas, percentage of village 




-0.004 




-0.040 


population 




(0.074) 




(0.051) 


Village is much or slightly poorer than other 




0.049 




-0.002 


villages in district 




(0.035) 




(0.026) 


Village has similar income level as other 




-0.026 




-0.017 


villages in district 




(0.032) 




(0.025) 


District is much or slightly poorer than other 




-0.049* 




-0.022 


districts 




(0.028) 




(0.017) 


RT head knows each community member very 




-0.017 




-0.037** 


well 




(0.025) 




(0.014) 


RT head is elected 




-0.071*** 
(0.023) 




-0.032** 
(0.014) 


Log distance village and district capital 




0.001 
(0.032) 




0.004 
(0.019) 


Constant 


0.288*** 
(0.064) 


0.308** 
(0.151) 


0.279*** 
(0.040) 


0.328*** 
(0.098) 




2,443 


2,443 


5,680 


5,680 


R- squared 


0.128 


0.137 


0.135 


0.138 


Adjusted R-squared 


0.123 


0.129 


0.132 


0.134 



" RW: ruhm warga (community association) 

Notes: This table reports the results of OLS estimates of the logarithm of the ratio PCE/PMT. All regressions include district fixed effects, 
and standard errors are clustered at the subdistrict level. Columns 1 and 2 are conditional on registration in the UDB and use the actual PMT 
score from the UDB. Columns 3 and 4 are unconditional on registration in the UDB, and are based on the PMT score reconstructed using 
Suseti variables. Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1 
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Turning to community-level characteristics, conditional on registration in the UDB, the RT head 
considering the district to be poorer than other districts is associated with exclusion (i.e., having an 
actual per capita consumption level about 5% lower than predicted by the PMT). This is consistent 
with the fact that OLS-based PMT ranking is less accurate at the left of the consumption distribution 
(Grosh and Baker 1995), making the distinction among relatively poor households more difficult. 37 Also 
consistent with this premise is the correlation of the RT head election with a PMT score higher than 
actual consumption, both conditional and unconditional on registration in the UDB. Indeed, although 
each district government has produced specific regulations regarding the organisation of neighbourhood 
associations since 1999 (Kurasawa 2009), RT heads are more often elected in rural areas, which are 
characterised by a higher poverty incidence. 



24 



37 Part of the literature advocates using alternative statistical approaches, given the inaccuracy of OLS-based PMT estimations 
at the bottom of the distribution. Muller and Bibi (2010), for instance, provide evidence that censored quantile regressions 
anchored on the first decile minimise undercoverage. 



6. Concluding Remarks 



This paper evaluates the effectiveness of the world's largest targeting system for delivering social 
programme benefits. We show that use of the UDB in Indonesia can be expected to significantly reduce 
leakage of benefits to non-poor households. However, undercoverage remains relatively high, due largely 
to the difficulties of enumerating the right households for inclusion in the UDB. We indeed predict a 
decrease in undercoverage by about 30% under a simulation that considers enumerating and estimating 
PMT scores for all households (as in a census), compared with focusing only on households that have 
been registered in the UDB. This is the first paper to identify the relative importance of enumeration 
versus PMT errors in determining the overall effectiveness of a large-scale targeting system. 

About 25 million households were surveyed for the UDB using a combination of poverty maps and 
suggestions from communities at a relatively low cost of around US$2 per household. Based on the 
finding that targeting accuracy may be improved through more extensive coverage of households, we 
propose two practical strategies to achieve this for Indonesia's UDB, as well as for other countries 
developing similar national targeting registries. 

The first proposed strategy is to increase the number of households enumerated in the national targeting 
registry survey. One option is to conduct a census of the full population (rather than only select 
households expected to be poor), as we simulate in this paper. Although it is commonly argued that it is 
too expensive to visit the entire population, we find that surveying the remaining 60% of the population 
would (only) cost about 11% of the value of additional benefits that would be received by households 
from the poorest three deciles nationally, assuming a one-third improvement in undercoverage and that 
the data collected are used for three years. If conducting a full census is nevertheless considered cost 
prohibitive, a related option would be to first identify the poorest areas based on small-area poverty 
maps and then survey all households in these geographic areas. 

The second proposed strategy, which could be combined with the first, is to transform the household 
targeting system registration into a more open process in order to allow greater entry. Other countries, 
such as Colombia and the Philippines, combine a complete census in the poorest areas with on-demand 
applications (also referred to as self-targeting) in other areas, in an attempt to survey as many poor 
households as possible while maintaining relatively low total registration costs. In an on-demand 
approach, households that consider themselves eligible for a given programme are allowed to apply 
for inclusion in the registry. In a randomised pilot experiment in Indonesia, Alatas et al. (20 1 3b) found 
that self-targeting leads to similar undercoverage as full enumeration at a lower overall cost, because it 
surveys fewer households. However, further research is needed to assess the cost-effectiveness of these 
different strategies, especially given evidence from this study that self-targeting excludes some of the 
poorest households, which do not apply — and leads to higher costs incurred directly by households 
(Alatas et al. 2013b). 

Either of these strategies could be further bolstered with a greater focus on improving the cost- 
effectiveness of the household registration process. For instance, one cost-effective alternative may 
be to shorten the targeting questionnaire to allow a larger number of households to be surveyed at a 
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lower cost, which might not necessarily come at the expense of targeting accuracy. Indeed, Bah (2013) 
showed that increasing the number of indicators included in a PMT formula from 10 to 30 does not 
significantly increase the accuracy of predicted household poverty levels nor reduce targeting errors. 38 

Furthermore, the targeting accuracy results in this paper are based on the assumption of perfect (or 
strong) correspondence between the beneficiary lists from the UDB registry and the households who 
actually end up receiving social programme benefits. A growing literature examining the political 
economy of targeting has provided evidence that, in practice, official beneficiary lists may be modified 
in the field, which may positively or negatively affect targeting outcomes. 39 In Indonesia, past research 
suggests some departures although overall adherence to official beneficiary lists has typically been 
quite high (World Bank 2012). Such targeting rule violations may prove beneficial if they allow the 
community to exert their greater ability to identify the very poor (Alatas et al. 2012) or if the capture 
of programme benefits by local elites is limited or generates relatively small welfare losses (Alatas et 
al. 2013a). Future research should combine the methods for evaluating targeting effectiveness that we 
advocate in this paper with an assessment of eligibility adherence in the field to identify which effects 
prevail overall. 



38 Dreze and Khera (2010) go even further by proposing the use of simplified targeting criteria so that "every household can 
attribute its inclusion in, or exclusion from, the list to a single criterion". Results from Niehaus and Atanassova (20 13) justify 
this argument: increasing the number of poverty indicators used to assess household socioeconomic status can have adverse 
effects in terms of targeting outcomes as it makes eligibility less transparent and therefore more subject to manipulation by 
corrupt agents at the local level. 

39 Camacho and Conover (2011) identify manipulation of eligibility scores by local officials, especially around the time of 
local elections in Colombia. In the case of India's BPL cards, Niehaus and Atanassova (2013) provide evidence of targeting 
rule violations by local officials that are due to corrupt behaviour. They argue that the addition of poverty indicators into 
targeting formulas undermined targeting effectiveness. As a result, BPL's 'de facto' allocation is much less progressive than 
the 'de jure' allocation, as the use of more poverty indicators makes eligibility less transparent and thereby facilitates violations 
of the official rules. 
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Appendix 



Table Al: List of Household- and Community-Level Explanatory Variables 



Variables 


Mean 


SD Min Max 


Panel A: Household-Level Variables 


Hidden assets 


Asset: Land. 


85.34 




Asset: Jewellery or gold/savings > Rp 500,000. 


44.30 




Asset: Livestock with value > Rp 500,000. 


22.26 




Exposure to shock 


Household experienced shock in past year. 


24.08 




Social connectedness 


Nb of family members living in the same RWa/RT. 


2.36 


2.26 0.00 20.00 


At least one household member migrated. 


40.87 




At least one household member participates in community group. 


78.85 




Position within the community 


Assistance from nongovernmental institution in past year. 


86.73 




Reception of zakat, past year. 


34.95 




(Log) per capita value of total nongovernmental assistance in past 
year. 


8.09 


3.31 0.00 15.43 


Panel B: Community-Level Variables 


Community relative economic status 


UDB enumeration quotas, percentage of village population. 


0.51 


0.21 0.04 0.99 


Village is much or slightly poorer than other villages in district. 


48.24 




Village has similar income level as other villages in district. 


39.05 




District is much or slightly poorer than other districts. 


50.07 




Potential for elite capture 


(Log) distance between village and district capital. 


3.58 


0.59 1.10 5.25 


RT head knows each community member very well. 


72.91 




RT head is elected. 


53.29 




a RW: rukun warga (community association). 

Notes: At the household level, partially hidden assets considered include land, jewellery, savings, and livestock of a value larger than Rp 
500,000 (about US$40). The shock variable is a dummy equal to one if the household has experienced the death or serious illness of a house- 
hold member, loss of employment, harvest or business failure, or a natural disaster between the baseline and end-line Suseti surveys (i.e., 
between January 201 1 and February 2012). Social connectedness variables include the number of other family members living in the same 
neighbourhood, as well as dummy variables for whether at least one household member migrated and participates in a community group. 



Variables translating household's socioeconomic position as viewed by the community include the reception of assistance from the nongovern- 
mental institutions and its per capita value, and the reception of zakat (religious assistance distributed through the mosque). The reception of 
zakat is also included in the nongovernmental assistance dummy (and in the annual per capita value), which excludes disaster-related assistance 
but includes assistance received from religious or political institutions as well as from national and international nongovernmental organisations, 
firms/corporations, and other private donors. At the community level, indicators of the relative position of the community include the UDB 
enumeration quotas as a share of total village population, as well as RT heads' perceptions on whether the village in which the neighbourhood is 
located is poorer or similar to other villages within the district, and whether the district is poorer than other districts in the province. Indicators 
of the potential or elite capture include the (logarithm) of the distance between the village and the district capital, from the 20 1 1 Village Census 
survey (Potensi Desa or PODES) conducted in Indonesia every three years, and dummies equal to one if the RT head is elected and if he/she 
declares that s/he knows each member of the community very well. 
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countries. This paper provides the first attempt to identify the relative importance of two key design issues 
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rank surveyed households. We evaluate the performance of Indonesia's Unified Database for Social Protection 
Programmes (UDB), the largest targeting registry in the world, which is used to provide social assistance to 
more than 25 million households. Linking administrative data with an independent household survey, we find 
that the UDB system is more progressive than previous targeting approaches used in Indonesia, leading to 
a decrease in benefit leakage to non-poor households. However, if poor households are not surveyed in the 
first place, even a perfect ranking method cannot prevent their exclusion. Under a simulation that considers 
enumerating and estimating proxy-means testing (PMT) scores for all households (as in a census), we estimate 
a one-third decrease in undercoverage compared to focusing on households that have been registered in the 
UDB. Investigating household- and community-level correlates of misenumeration and misclassification, we find 
evidence that local communities use different definitions of poverty and have better information on the welfare 
status of their members. 



TIM NASIONAL PERCEPATAN PENANGGULANGAN KEMISKINAN 
(TNP2K) 



Jl. Kebon Sirih No. 35, Jakarta Pusat 10110 



E-mail: 



+62(0) 213912812 
+62 (0)213912511 
info@tnp2k.go.id 
www.tnp2k.go.id 




^ Printed on recycled paper 



ISBN 9786022751267 



786022 11 751267 



